Generalization of ERM in Stochastic Convex Optimization: The Dimension Strikes Back

In stochastic convex optimization the goal is to minimize a convex function F (x) . = Ef∼D[f(x)] over a convex set K ⊂ R where D is some unknown distribution and each f(·) in the support of D is convex over K. The optimization is commonly based on i.i.d. samples f, f, . . . , f from D. A standard approach to such problems is empirical risk minimization (ERM) that optimizes FS(x) . = 1 n ∑ i≤n f (x). Here we consider the question of how many samples are necessary for ERM to succeed and the closely related question of uniform convergence of FS to F over K. We demonstrate that in the standard `p/`q setting of Lipschitz-bounded functions over a K of bounded radius, ERM requires sample size that scales linearly with the dimension d. This nearly matches standard upper bounds and improves on Ω(log d) dependence proved for `2/`2 setting in [SSSS09]. In stark contrast, these problems can be solved using dimension-independent number of samples for `2/`2 setting and log d dependence for `1/`∞ setting using other approaches. We also demonstrate that for a more general class of rangebounded (but not Lipschitz-bounded) stochastic convex programs an even stronger gap appears already in dimension 2.